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generally. Finally, the content of programs was coded as addressed to 
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explicit instruction methods. Intensive programs proved more 
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Critical thinking, the ability and willingness to test the 
validity of propositions, has long been held as a desirable 
product of education. Liberal arts education, for example, is 
often defended for its presumed capacity to develop adaptable 
thinkers to meet any variety of societal challenges. In a 1984 
report, the National Institute of Education identified critical 
thinking as one of the highest priorities of college education in 
general (as cited in McMillan, 1987) . Many professions, frorn 
nursing to law to science, identify critical thinking as a crucial 
skill of its practice. And critical thinking is regarded by many 
as a fundamental quality of good citizens of democracy. 

There is no such agreement, however, about the exact nature of 
this ability. Critical thinking has been equated with 
intelligence, domain-specific expertise, problem-solving, reading 
comprehension, logic and sound reasoning, and other higher order 
mental activities. This disagreement reflects the complexity of 
this construct. Critical thinkers must have sufficient 
information about a proposition to permit its evaluation. 
Critical thinkers must also be self-evaluative, knowing when their 
information is incomplete or faulty and where to find reliable 
sources of information. Critical thinkers must have an arsenal of 
analytical, problem-solving, and inquiry skills available to 
decompose complex arguments or information into simpler 
components, generate hypotheses and questions about the 
components, and then submit these components to careful scrutiny. 
Finally, critical thinking has an attitudinal aspect. That is, 
critical thinking is not merely problem-solving algorithm, but is 



an inquisitive disposition toward experience, a predisposition to 
intellectually act in a consistent way over time and in different 
situations . 

Given this complex of declarative, procedural, meta-cognitive, 
and attitudinal aptitudes, one would expect that critical thinking 
would develop only after years of sustained educational effort. 
Yet there are numerous studies of the effects of relatively brief 
interventions on critical thinking. The results of these programs 
have been mixed. Ennis (1989) categorizes instruction for 
critical thinking into 3 models: general, infusion, and immersion. 
The general model attempts to explicitly teach general critical 
thinking skills. The infusion model may also explicitly teach 
critical thinking skills, but in the specific context of some 
subject matter domain. The immersion model suggests that critical 
thinking is best developed by encouraging a deep understanding of 
a subject matter domain, not by explicit instruction. According 
to Ennis, there is not enough evidence yet to suggest that any one 
of these approaches is better than the other, though he expresses 
some reservations about the rationale underlying the immersion 
model . 

McMillan (1387) examined 27 evaluations of educational effects 
on college students' critical thinking. He concluded, "The 
results failed to support the use of specific instructional or 
course conditions to enhance critical thinking but did support the 
conclusion that college in general appears to improve critical 
thinking." McKeachie, Pintrich, Lin, and Smith (1987) disagreed 
with McMillan's assessment. They attributed improvement in 
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critical thinking to three instructional variables: student 
discussion, explicit emphasis on problem-solving procedures, and 
explicit amphasis on methods to encourage development of raeta- 
cognition . 

Nisbett and his colleagues (Nisbett, Fong, Lehaman, & Cheng, 
1987) have suggested a very different kind of explicit instruction 
for the promotion of critical thinking. They have argued that 
higher order thinking is neither task-specific nor formally 
logical, but the result of repeated practical experiences with 
situations that demand certain patterns of thinking. These 
abstracted patterns of well-practiced cognition are called 
pragmatic reasoning schemas. Nisbett argues that pragmatic 
reasoning schemas can be enhanced by relatively brief, explicit 
instruction about the schema. 

We employed meta-analysis to synthesize the results of explicit 
instruction on critical thinking. We hoped to obtain an accurate 
impression of the overall success of such programs and, hopefully, 
some insight into specific instructional features that might be 
most effective. We used study effect meta-analysis (Bangert- 
Drowns, 1986),. an adaptation of the meta-analytic techniques 
originally proposed by Gene Glass (Glass, McGaw, & Smith, 1981) . 
This approach involves four phases: the collection of studies, 
coding of the study features, calculation of effect sizes, and 
examination of relations between study features and effect sizes. 
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Method 



We searched the keywords "critical thinking" with "instruction" 
and "research" in the SRIC database and Dissertation Ahst r^^^^t'l 
Internflt.innal . As we obtained useful reviews and studies, we 
examined their bibliographies for leads to other studies. In all, 
we examined approximately 250 books, articles, dissertations, and 
abstracts. Many of these were conceptual or philosophical 
articles, others were correlational studies, still others 
investigated educationally relevant variables that were not 
specific to instruction. 

To be included in the meta-analysis, studies had to compare the 
performance of two groups of students on some measure of critical 
thinking. One group received instruction to improve critical 
thinking, the other did not. The instructional intervention had 
to be explicit instruction. That is, the concepts or skills to be 
taught were explicitly labelled and discussed, and students were 
led through practice on the use of this content. Of course, 
studies also had to report sufficient information for the 
calculation of effect sizes. 

Each study was coded for its characteristics in three 
categories: publication history (study source and publication 
date), methodological and setting features ^e.g., method of 
subject assignment, use of same or different teachers in 
experimental and control treatments, grade level) , and 
instructional features. Two instructional variables were the 
duration of instruction and the subject matter of the course in 



which the instruction was delivered. Another variable recorded 
whether instruction was given by the teacher or through some 
media. Critical thinking programs that continuously focussed on 
thinking skills were distinguished from those who only 
periodically addressed them. Programs that taught critical 
thinking in a subject matter domain were distinguished from those 
that taught critical thinking generally. Finally, the content of 
the critical thinking program was coded as either addressed to 
internal consistency (examining the quality of logic, reasoning, 
or argumentation), external consistency (examining the quality of 
evidence for a proposition) , or both. 

Effect sizes were calculated for measures of critical thinking. 
An effect size is the difference between experimental and control 
group means divided by the standard deviation of the control 
group. Where means and standard deviations were not available, 
techniques described in Glass, McGaw, and Smith (1981) were used 
to compute effect sizes from other available statistics, like £- 
and L.- values. To preserve the independence of the datapoints, 
only one effect size was calculated for each study. 

Results 

Twenty studies were found to be suitable for this meta- 
analysis. Nineteen of these studies were doctoral dissertations- 
All used a general measure of critical thinking as the final 
criteria. Ten used the Watson-Glaser Critical Thinking Appraisal, 
and five used the Cornell Critical Thinking Test. 



The most obvious result of this review was the general success 
of explicit treatment to produce improvement on tests of critical 
thinking. Findings from eighteen of the twenty studies favored 
critical thinking instruction, eight significantly. Only two 
studies produced negative findings, and these were nonsignificant. 
The average effect size was statistically and practically 
significant, 0.37 standard deviations. This is equivalent to 
increasing student performance from the 50th to the 64th 
percentile . 

Only one variable was found to be significantly related to 
effect size. When the content of the critical thinking 
instruction was internal validity (logic, reasoning, and the 
like), the average effect size (0.03 standard deviations) was 
significantly lower than when both internal and external validity 
were emphasized (0.55 standard deviations). Programs that 
emphasized external validity produced an intermediate effect (0.40 
standard deviations) , but this finding was based on only three 
studies . 

Two other variables were related to effect size with borderline 
significance (ji-.ll). Student in younger grades benefitted more 
from critical thinking instruction than did students in high 
school and college. The average effect size for elementary and 
junior high grades was 0.50 standard deviations, for high school 
and college, 0.21 standard deviations. We also found that 
programs that concentrated continuously on critical thinking were 
more successful than programs which periodically emphasized 
critcai thinking. Continuous programs produced an average effect 



size of 0.47 standard deviations. Programs that periodically 
addressed critical thinking produced an average effect of 0.15 
standard deviations. 



Discussion 



It is most striking that the studies reviewed in this meta- 
analysis so consistently produced findings favorable to explicit 
instruction in critical thinking. Eighteen- of twenty studies 
produced positive findings, and all twenty yielded a respectable 
effect size of 0.40 standard deviations. Of course, this does not 
necessarily mean that critical thinking itself is so easily 
increased. Though instructed students did better than 
uninstructed students on paper-and-pencil tests of general 
critical thinking ability, they may not be able or willing to use 
these skills in natural, everyday situations. Even so, that 
relatively brief interventions could result in such noticeable 
improvement in critical thinking test performance is impressive. 

Six of the studies employed instructional programs that focused 
primarily on assessing the internal validity (logic, reasoning, 
etc.) of propositions. Four of these taught logic through 
explicit instruction, two concentrated on improving performance on 
test items typically used to measure thinking and intelligence. 
These programs proved least successful in promoting critical 
thinking skill. Students of explicit instruction that addressed 
matters of both internal and external validity, on the other hand, 
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performed half a standard deviation better than their uninstructed 
counterparts. 

Substantial evidence indicates that humans do not use formal 
logic to solve everyday problems, and therefore teaching logic may 
be too obscure and seemingly irrelevant to enhance critical 
thinking. On the other hand, Nisbett, Fong, Lehman, and Cheng 
(1987) argued that people seem to possess relatively general 
informal thinking skills that are presumably generalized from 
repeated experience with situations of particular types. Cheng, 
Holyoak, Nisbett, and Oliver (198S) showed that these pragmatic 
reasoning schemas could be enhanced with even short instructional 
interventions . 

Assessing the trustworthiness of a statement, that is, its 
external validity, is a common demand of even young children's 
lives. This meta-analysis showed that instruction in this 
pragmatic reasoning skill, even when combined with instruction on 
logic, is more effective than teaching formal logic alone. The 
existence of pragmatic skills for assessing external validity may 
explain why children in younger grades seem to especially benefit 
from the instruction. High school and college students have 
practiced these pragmatic skills for a longer uf.me and instruction 
might be expected to have a smaller influence on their thinking 
than on younger, less experienced students. 

We found tentative evidence that more intensive programs, ones 
that continuously emphasize critical thinking, do better than 
programs that only periodically address critical thinking. 
Interestingly, we found no evidence that length of treatment is 

9 

o 10 

ERIC 



i 



related to the effects of treatment. It is also worth noting that 
critical thinking programs embedded in specific subject matter 
instruction (what Ennis [1989] calls "infusion" programs) do no 
better than general explicit instruction for critical thinking. 
Of course, the tests used in these studies were measures of 
general critical thinking. It is possible that the effects of 
content -embedded critical thinking instruction would appear only 
on tests requiring application to the content. 
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